Image processing methods and systems for barcode and/or product label recognition

ABSTRACT

This disclosure provides an image processing method and system for recognizing barcodes and/or product labels. According to an exemplary embodiment, the method uses a multifaceted detection process that includes both image enhancement of a candidate barcode region and other product label information associated with a candidate barcode region to identify a product label, where the candidate barcode region includes a nonreadable barcode. According to one exemplary application, a store profile is generated based on the identifications of the product labels which are associated with a location of a product within a store.

CROSS REFERENCE TO RELATED PATENTS AND APPLICATIONS

Cross-reference is made to the following copending applications, filedcontemporaneously herewith: U.S. patent application Ser No. ______,filed ______, by Wu et al., and entitled “STORE SHELF IMAGING SYSTEM”and U.S. patent application Ser. No. ______, filed ______, by Wu et al.,and entitled “METHOD AND SYSTEM FOR SPATIAL CHARACTERIZATION OF ANIMAGING SYSTEM”, the disclosures of which are incorporated herein byreference in their entireties.

BACKGROUND

This disclosure provides an image processing method and system forrecognizing barcodes and/or product labels. According to an exemplaryembodiment, the method uses a multifaceted detection process thatincludes both image enhancement of a candidate barcode region and otherproduct label information associated with a candidate barcode region toidentify a product label, where the candidate barcode region includes anonreadable barcode. According to one exemplary application, a storeprofile is generated based on the identifications of the product labelswhich are associated with a location of a product within a store.

This disclosure also relates to product mapping and finds particularapplication in connection with a system and method for determining thespatial layout of product content of a product facility, such as astore.

Retail chains, such as pharmacy, grocery, home improvement, and others,may have a set of product facilities, such as stores, in which productsare presented on product display units, such as shelves, cases, and thelike. Product information is generally displayed close to the product,on preprinted product labels. The product labels indicate the price ofthe item and generally include a unique identifier for the product,e.g., in the form of a barcode, which is often used by the store forrestocking and other purposes. Periodically, stores place some of theitems on sale, or otherwise adjust prices. This entails printing of saleitem labels and/or associated signage and manual replacement of theproduct labels and/or addition of associated signage. The printing andposting of such sale item signage within each store often occurs atweekly intervals.

It would be advantageous to each store if the signage was printed andpacked in the order in which a store employee encounters the saleproducts while walking down each aisle. However, retail chains generallycannot control or predict the product locations across each of theirstores. This may be due to a number of factors, such as store managerdiscretion, local product merchandising campaigns, different storelayouts, and so forth. Thus, individual stores may resort to manuallypre-sorting the signage into the specific order appropriate for thatstore, which can be time consuming and not always accurate.

Copending patent applications U.S. patent application Ser. No. ______,filed ______, by Wu et al., and entitled “Store Shelf Imaging System”and U.S. patent application Ser. No. ______, filed ______, by Wu et al.,and entitled “Method and System for Spatial Characterization of ImagingSystem” provide a method and system for a chain of stores to be able tocollect product location data automatically across its stores. Eachstore could then receive signage which has been automatically packagedin an appropriate order to avoid a pre-sorting step.

There exist many prior arts on barcode detection and/or recognition, seePéter Bodnár and László G. Nyúl, “Improving Barcode Detection withCombination of Simple Detectors,” 2012 Eighth International Conferenceon Signal Image Technology and Internet Based Systems (2012) and J.Liyanage, “Efficient Decoding of Blurred, Pitched, and Scratched BarcodeImages,” Second International Conference on Industrial and InformationSystems (ICIIS 2007), August, (2007), and citations of them. They canperform quite well with sufficient image resolution and high imagequality (no motion blur, no out of focus, good and uniform illumination. . . ). In practice, high quality imaging is not always feasible oraffordable. As a result, barcode recognition is still a fairly activeresearch area focusing on solving real-world problems even though it mayseem straightforward. See Péter Bodnár and László G. Nyúl, “ImprovingBarcode Detection with Combination of Simple Detectors,” 2012 EighthInternational Conference on Signal Image Technology and Internet BasedSystems (2012) and J. Liyanage, “Efficient Decoding of Blurred, Pitched,and Scratched Barcode Images,” Second International Conference onIndustrial and Information Systems (ICIIS 2007), August, (2007). For aretail application as disclosed in U.S. patent application Ser. No.______, filed ______, by Wu et al., and entitled “Store Shelf ImagingSystem”, high throughput and broad spatial coverage, i.e., the entirestore, are required where 15000 barcodes or more covering the entirestore need to be recognized in a relatively short time-frame, e.g., 4-8hours. This makes the matter worse since maintaining high qualityimaging over a large spatial area while achieving such throughput is nota simple task. Hence improvement on existing barcode detection andrecognition methods is needed.

INCORPORATION BY REFERENCE

Adelmann et al., “Toolkit for Bar Code Recognition and Resolving onCamera Phones—Jump-Starting the Internet of Things”, Informatik Workshopon Mobile and Embedded Interactive Systems, 2006;

Bailey, “Super-Resolution of Bar Codes”, Journal of Electronic Imaging,Vol. 10, No. 1, 2001, p 213-220;

Ballard, “Generalizing the Hough Transform to Detect Arbitrary Shapes”,Pattern Recognition, Vol. 13, No. 2, 1981, p 111-122;

Bodnár et al, “Barcode Detection With Uniform Partitioning andMorphological Operations”, Conf. of PhD Students in Computer Science,2012, p 4-5;

Bodnár et al, “Efficient Barcode Detection With Texture Analysis”,Proceedings of the Ninth IASTED Int'l Conf. on Signal Processing,Pattern Recognition, and Applications”, 2012, p 51-57;

Bodnár et al, “Improving Barcode Detection With Combination of SimpleDetectors”, Int'l Conf. on Signal Image Technology and Internet BasedSystems, 2012;

Canny, “A Computational Approach to Edge Detection”, IEEE Transactionson Pattern Analysis and Machine Intelligence, Vol. PAMI-8, No. 6, 1986,p 679-698;

Ebner et al., “Development and Testing of a Color Space (IPT) WithImproved Hue Uniformity”, Proceedings of IS&T/SID's Sixth Color ImagingConf., 1998, p. 8-13;

Felzenszwalb et al., “Distance Transforms of Sampled Functions”, CornellComputing and Information Science, Tech. Rep., 2004.

Gonzalez et al., “Digital Image Processing”, 3rd Edition, Prentice Hall,2008;

Hodges, “An Introduction to Video and Audio Measurement”, Elsevier,2004, p 174;

Hunt, “The Reproduction of Colour”, John Wiley & Sons, 2004, p 48;

Jain et al., “Bar Code Localization Using Texture Analysis”, Proceedingsof the Second Int'l Conf. on Document Analysis and Recognition, 1993, p41-44;

Joseph et al., “Bar Code Waveform Recognition Using Peak Locations”,IEEE Transactions on Pattern Analysis and Machine Intelligence, Vol. 16,No. 6, 1994, p 630-640;

Juett, “Barcode Localization Using a Bottom Hat Filter”, NSF ResearchExperience for Undergraduates, 2005;

Katona et al., “A Novel Method for Accurate and Efficient BarcodeDetection With Morphological Operations”, Eighth Int'l Conf. on SignalImage Technology and Internet Based Systems, 2012, p 307-314;

Kiryati et al., “A Probabilistic Hough Transform”, Pattern Recognition,Vol. 24, No. 4, 1991, p 303-316;

Kuroki et al., “Bar Code Recognition System Using Image Processing”,Hitachi Process Computer Engineering, Inc., 1990;

Lin et al., “Multi-Symbology and Multiple 1D/2D Barcodes ExtractionFramework”, Advances in Multimedia Modeling, 2011, p 401-410;

Lin et al., “Real-Time Automatic Recognition of Omnidirectional MultipleBarcodes and DSP Implementation”, Machine Vision and Applications, Vol.22, 2011, p 409-419;

Liyanage, “Efficient Decoding of Blurred, Pitched, and Scratched BarcodeImages”, Second Int'l Conf. on Industrial and Information Systems, 2007;

McKesson, “Linearity and Gamma—Chapter 12—Dynamic Range”,http://www.arcsynthesis.org/gItut/Illumination/Tut12%20Monitors%20and%20Gamma.html, retrieved Jul. 11, 2013;

Muniz et al., “A Robust Software Barcode Reader Using the HoughTransform”, Int'lConf. on Information Intelligence and Systems, 1999, p313-319;

Normand et al., “A Two-Dimensional Bar Code Reader”, 12th Int'l Conf. onPattern Recognition, Vol. 3, 1994, p 201-203;

Ohbuchi et al., “Barcode Readers Using the Camera Device in MobilePhones”, Proceedings of the 2004 Int'lConf. on Cyberworlds, 2004;

Oktem et al., “A Superesolution Approach for Bar Code Reading”,Electrical and Engineering Department, Atilim University, Turkey, 2002;

Oktem, “Bar Code Localization In Wavelet Domain By Using Binary”,Proceedings of the IEEE 12th Signal Processing and CommunicationsApplications Conference, 2004, p 499-501;

Pavlidis et al., “Fundamentals of Bar Code Information Theory”, IEEETransactions on Pattern Analysis and Machine Intelligence, Vol. 16, No.6, 1994, p. 630-640;

Poynton, “Digital Video and HDTV: Algorithms and Interfaces” MorganKaufman Publishers, 2003, p 260, 630;

Poynton, “Frequently Questioned Answers About Gamma”,www.poynton.com/notes/color/GammaFQA.html , 2010;

Reinhard et al., “High Dynamic Range Imaging: Acquisition, Display, andImage-Based Lighting”, Morgan Kaufmann Publishers, 2010, p 82.

Tuinstra, “Reading Barcodes From Digital Imagery”, Ph.D. dissertation,Cedarville University, 2006;

Wittman et al., “Super-Resolution of 1D Barcode Images”, University ofMinnesota, 2004;

Wu et al., “Automatic Thresholding of Gray-Level Using Multi-StageApproach”, Proceedings of the Seventh Int'l Conf. on Document Analysisand Recognition, 2003, p 493-497;

Youssef et al., “Automated Barcode Recognition for Smart Identificationand Inspection Automation”, Expert Systems with Applications, Vol. 33,No. 4, 2007, p 968-977;

Zhang, “Flexible Camera Calibration By Viewing a Plane From UnknownOrientations”, Int'lConf. on Computer Vision, Corfu, Greece, 1999, p666-673;

U.S. Pat. No. 5,512,739, granted on Apr. 30, 1996, to Chandler et al.and entitled “DUAL PROCESSOR OMNIDIRECTIONAL BAR CODE READER WITH DUALMEMORY FOR BAR CODE LOCATION AND ORIENTATION”;

U.S. Pat. No. 7,066,291, granted on Jun. 27, 2006, to Martins et al. andentitled “ROBOT SYSTEM”;

U.S. Pat. No. 7,290,707, granted Nov. 6, 2007, to Sawasaki and entitled“TELE-INVENTORY SYSTEMS AND IN-SHOP TERMINAL AND REMOTE MANAGEMENTAPPARATUS FOR THE SYSTEM”;

U.S. Pat. No. 7,386,163, granted Jun. 10, 2008, and entitled. “OBSTACLERECOGNITION APPARATUS AND METHOD, OBSTACLE RECOGNITION PROGRAM, ANDMOBILE ROBOT APPARATUS”;

U.S. Pat. No. 7,693,757, granted Apr. 6, 2010, to Zimmerman and entitled“SYSTEM AND METHOD FOR PERFORMING INVENTORY USING A MOBILE INVENTORYROBOT”;

U.S. Pat. No. 8,189,855, granted May 29, 2012, to Opalach et al. andentitled “PLANOGRAM EXTRACTION BASED ON IMAGE PROCESSING”;

U.S. Published Patent App. No. 2013-0342706, published on Dec. 26, 2013,to Hoover et al. and entitled “CAMERA CALIBRATION APPLICATION”;

U.S. Published Patent App. No. 2013-0229517, published on Sep. 5, 2013to Kozitsky et al. and entitled “VEHICLE SPEED MEASUREMENT METHOD ANDSYSTEM UTILIZING A SINGLE IMAGE CAPTURING UNIT”;

U.S. Published Patent App. No. 2013-0342706, published on Dec. 26, 2013,to Hoover et al. and entitled “CAMERA CALIBRATION APPLICATION”;

U.S. Published Patent App. No. 2013-0278761, published on Oct. 24, 2013,to Wu and entitled “REAL-TIME VIDEO TRIGGERING FOR TRAFFIC SURVEILANCEAND PHOTO ENFORCEMENT APPLICATIONS USING NEAR INFRARED VIDEOACQUISITION”, are incorporated herein by reference in their entirety.

BRIEF DESCRIPTION

In one embodiment of this disclosure, described is a method ofperforming decoding of a barcode associated with a product label, theproduct label including one or more barcodes and other product labelinformation, the method comprising: an image capturing device capturingan image of the product label and storing the captured image in a memoryoperatively associated with the image capturing device; a processoroperatively associated with the memory detecting and localizing one ormore barcode candidate regions within the captured image of the productlabel, the barcode candidate regions including a substantially fewernumber of pixels relative to a total number of pixels included in thecaptured pixel image of the product label; cropping the detected andlocalized one or more barcode candidate regions to generate one or moresub-images including images of the one or more barcode candidateregions; processing each sub-image using two or more independent imagequality improvement processes to generate modified versions of thebarcode candidate regions; and processing the modified versions of thebarcode candidate regions to decode readable barcodes included in themodified versions of the candidate barcode regions.

In another embodiment of this disclosure, described is an imageprocessing system for performing decoding of a barcode associated with aproduct label, the product label including one or more barcodes andother product label information, the image processing system comprising:an image capturing device; and a processor operatively connected to theimage capturing device, the processor configured to: the image capturingdevice capturing an image of the product label and storing the capturedimage in a memory operatively associated with the image capturingdevice; the processor operatively associated with the memory detectingand localizing one or more barcode candidate regions within the capturedimage of the product label, the barcode candidate regions including asubstantially fewer number of pixels relative to a total number ofpixels included in the captured pixel image of the product label;cropping the detected and localized one or more barcode candidateregions to generate one or more sub-images including images of the oneor more barcode candidate regions; processing each sub-image using twoor more independent image quality improvement processes to generatemodified versions of the barcode candidate regions; and processing themodified versions of the barcode candidate regions to decode readablebarcodes included in the modified versions of the candidate barcoderegions.

In still another embodiment of this disclosure, described is a method ofperforming product label identification, the product label including oneor more barcodes and other product label information, the methodcomprising: an image capturing device capturing an image of the productlabel and storing the captured image in a memory operatively associatedwith the image capturing device; a processor operatively associated withthe memory detecting and localizing one or more barcode candidateregions within the captured image of the product label; cropping thedetected and localized one or more barcode candidate regions to generateone or more sub-mages including the one or more barcode candidateregions; processing each sub-image to decode readable barcodes includedin the barcode candidate regions and identify barcode candidate regionsincluding an unreadable barcode; processing all or a portion of thecaptured image of the product label associated with the unreadablebarcode to determine all or part of the other product label informationassociation with the unreadable barcode; comparing the determined otherproduct label information to a plurality of product label templates todetermine a layout associated with the product label including anunreadable barcode candidate region; processing the captured image ofthe product label to extract all or part of the other product labelinformation based on the determined layout associated with the productlabel; and identifying the captured image of the product label asincluding one of a plurality of unique predefined product labels.

In still yet another embodiment, disclosed is an image processing systemfor performing product label identification, the product label includingone or more barcodes and other product label information, the imageprocessing system comprising: an image capturing device; and a processoroperatively connected to the image capturing device, the processorconfigured to: the image capturing device capturing an image of theproduct label and storing the captured image in a memory operativelyassociated with the image capturing device; the processor operativelyassociated with the memory detecting and localizing one or more barcodecandidate regions within the captured image of the product label;cropping the detected and localized one or more barcode candidateregions to generate one or more sub-images including the one or morebarcode candidate regions; processing each sub-image to decode readablebarcodes included in the barcode candidate regions and identify barcodecandidate regions including an unreadable barcode; processing all or aportion of the captured image of the product label associated with theunreadable barcode to determine all or part of the other product labelinformation association with the unreadable barcode; comparing thedetermined other product label information to a plurality of productlabel templates to determine a layout associated with the product labelincluding an unreadable barcode candidate region; processing thecaptured image of the product label to extract all or part of the otherproduct label information based on the determined layout associated withthe product label; and identifying the captured image of the productlabel as including one of a plurality of unique predefined productlabels.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a schematic elevational view of a store profile generationsystem in accordance with one aspect of the exemplary embodiment;

FIG. 2 is a schematic elevational view of a store profile generationsystem in accordance with another aspect of the exemplary embodiment;

FIG. 3 is a schematic elevational view of a store profile generationsystem in accordance with another aspect of the exemplary embodiment;

FIG. 4 is a schematic elevational view of a store profile generationsystem in accordance with another aspect of the exemplary embodiment;

FIG. 5 is a functional block diagram of the store profile generationsystem of FIGS. 1-4 in accordance with one aspect of the exemplaryembodiment;

FIG. 6 illustrates an exemplary price tag;

FIG. 7 is a flow chart illustrating a store profile generation method inaccordance with another aspect of the exemplary embodiment;

FIG. 8 is a flow chart illustrating a barcode and product labelrecognition method in accordance with one aspect of the exemplaryembodiment;

FIG. 9 illustrates an exemplary acquired image of a store shelf,including detected candidate barcode regions;

FIG. 10 is a flow chart illustrating a barcode region detection methodin accordance with one aspect of the exemplary embodiment; and

FIG. 11 illustrates exemplary price tags, i.e., product label.

DETAILED DESCRIPTION

This disclosure provides a method and system to improve the opticaldetection of barcodes in a multi-camera system that determines thelayout of a store. The method utilizes a multifaceted detection approachthat includes both image enhancement and auxiliary information toimprove on current optical barcode detection methods. The method firstemploys a simple method to detect and crop candidate barcodes from thecaptured camera image. These candidate regions are then analyzed by anoptical barcode reader; segments that are correctly read are used toupdate a store layout map. If a candidate area fails to be detected bythe optical reader the area is processed using several simple imageenhancements methods (e.g. gamma correction) and resubmitted to thereader. If the enhanced cropped areas still fails barcode detection, thearea around the code is processed for a-priori contextual information.Such information may include objects such as item description text orprice/sale signs (e.g. yellow or red price boxes). The contextualinformation is compared against a database to determine a product match.If a unique match is found, this is used to identify the product toupdate the store layout map. If non-unique matches are found, a list ofpossible products with corresponding confidence measures are provided asadditional information to the store layout map.

With reference to FIGS. 1-5, where the same numbers are used for similarelements, a mobile profile generation system 10 is configured fordetermining a spatial layout 12 (FIG. 5) of the product content of aproduct facility, such as a retail store, warehouse, or the like. Thespatial layout may be referred to herein as a store profile. The storeprofile 12 may be in the form of a 2-dimensional or 3-dimensional planof the store which indicates the locations of products, for example, byproviding product data for each product, such as an SKU or barcode, andan associated location, such as x,y coordinates (where x is generally adirection parallel to an aisle and y is orthogonal to it), a position onan aisle, or a position on a predefined path, such as a walking paththrough the store. In some embodiments, the store profile may include aphotographic panorama of a part of the store generated from a set ofcaptured images, or a graphical representation generated therefrom.

The store profile 12 is generated by capturing images of product displayunits 14, such as store shelf units, at appropriate locations withappropriate imaging resolutions. As illustrated in FIG. 1, each shelfunit 14 may include two or more vertically-spaced shelves 16, to whichproduct labels 18, such as product price tags, displayingproduct-related information, are mounted, adjacent related products 19.In the exemplary embodiments, the price labels are not on the productsthemselves, but on the shelf units, e.g., in determined locations. Thusfor example, a portion of a shelf which is allocated to a given productmay provide for one (or more) price labels to be displayed for thatproduct. In other embodiments the product labels 18 may be displayed onan adjacent pegboard or be otherwise associated with the respectivedisplay unit 14.

The exemplary profile generation system 10 includes a mobile base 20, animage capture assembly 22, and a control unit 24, which are moveable asa unit around the product facility. The exemplary system 10 capturesimages within a product facility, such as a retail store, with the imagecapture assembly 22 at a sequence of locations of the mobile base 20,extracts product-related data 26 (e.g., printed barcodes and/or textfrom the captured product price labels) and location information fromthe images and the mobile base location, and constructs a store profile12 (e.g., a 2D map, as discussed above) which defines a spatial layoutof locations of the shelf labels 18 within the store.

The mobile base 20 serves to transport the image capture assembly 22around the product facility and may be fully-autonomous orsemi-autonomous. In one embodiment, the mobile base 20 is responsiblefor navigating the system 10 to a desired location with desired facing(orientation), as requested by the control unit 24, and reporting backthe actual location and facing, if there is any deviation from therequest. As illustrated in FIG. 5, in a fully-autonomous mode, themotorized mobile base 20 may include a navigation component 30 and anassociated power source 32, such as a battery, motor, drive train, etc.,to drive wheels 34 of the of the mobile base in order to move the system10 to a desired location with desired facing according to a request fromthe control unit 24. The navigation component 30 may be similarlyconfigured to the control unit 24 and may include memory and a processorfor implementing the instructions provided by the control unit andreporting location and orientation information back to the control unit.Position and/or motion sensors 36 provide the navigation component 30with sensing capability to confirm and/or measure any deviation from therequested location and orientation. These may be used by the navigationcomponent for identifying the location, orientation, and movement of themobile base for navigation and for store profile generation by thecontrol unit. One suitable mobile base which can be adapted to useherein is a Husky™ unmanned ground vehicle obtainable from ClearpathRobotics Inc., 148 Manitou Dr., Kitchener, Ontario N2C 1L3, Canada,which includes a battery-powered power source.

In a semi-autonomous mode, the mobile base 20 is pushed by a person(e.g., as a cart), and thus the power source and optionally also thenavigation component may be omitted. In some embodiments, the navigationcomponent and sensors may be used in the semi-automated mode to confirmand/or measure any deviation from a requested location and orientation(e.g., by using voice feedback to confirm the aisle/shelf information orusing image features of the scene).

The image capture assembly 22 includes an imaging component 38 whichincludes one or more image capture devices, such as digital cameras 40,42, 44, that are carried by a support frame 46. The image capturedevices capture digital images, such as color or monochrome photographicimages. The support frame may be mounted to the mobile base 20 andextend generally vertically (in the z-direction) therefrom (for example,at an angle of from 0-30° from vertical, such as from 0-20° fromvertical). The cameras are configured to capture images of a full heighth of the shelf unit, or at least that portion of the height h in whichthe labels 18 of interest are likely to be positioned throughout thefacility.

One or more of the camera(s) 40, 42, 44 may be moveable, by a suitablemechanism, in one or more directions, relative to the support frame 46and/or mobile base 20. In one embodiment, at least one of the camerashas a first position and a second position, vertically-spaced from thefirst position, allowing the camera to capture images in the first andsecond positions. In the embodiment illustrated in FIGS. 2 and 3, forexample, the support frame 46 includes a translation stage 48 for movingone or more of the camera(s) in at least one direction, such asgenerally in the z (vertical) direction, as illustrated by arrow 49. Thedirection of movement need not be strictly vertical if the supporttranslation stage is mounted to an angled support frame, as noted above.Optionally, the translation stage 48 provides for rotation of one ormore of the cameras in the x, y plane and/or tilting of one or more ofthe cameras, relative to the translation stage/support frame. In anotherembodiment, the cameras, and/or their associated mountings, may providethe cameras with individual Pan-Tilt-Zoom (PTZ) capability. The pancapability allows movement of the field of view (FOV) relative to thebase unit in the x direction; the tilt capability allows the field ofview to move in the z direction as illustrated for camera 44 in FIG. 3;the zoom capability increases/decreases the field of view in the x, zplane (which may be measured in units of distance, such as inches or cm,as illustrated in FIG. 3, or angle a, as illustrated in FIG. 1). In someembodiments, only some, i.e., fewer than all, of the cameras aremoveable and/or have PTZ capability, as illustrated in FIG. 4, whereonly camera 42 has such capabilities. The incremental movement of themobile base 20 allows images to be captured along the length of theshelf unit 14 (in the x direction).

The image capture assembly 22 serves to capture a series of imagescontaining shelf product labels 18, such as product price tags, atsufficient resolution for analysis and product recognition. The productprice or tags 18 may be located on the outer edge of a shelf or at theend of a pegboard hook 50, or other product label mounting device. Asillustrated in FIG. 6, each price tag 18 generally includes a uniqueidentifier 54 for the product, such as a 1 or 2-dimensional barcode orstock keeping unit (SKU) code. As an example, a 1D EAN-13 code may beprinted on or otherwise affixed to the product label. 2D barcodes arecommonly referred to as QR codes or matrix codes. In addition, ahuman-readable price 56 and optionally some descriptive text 58 may beprinted on or otherwise affixed to the product label.

A width w of the barcode 54 in the y direction may be about 20-25 mm onmany price tags. However, the barcode width may not be uniformthroughout the store or from one store to another. In order to allowaccurate imaging and decoding of such barcodes, a minimum resolution ofapproximately 200 pixels per inch (ppi) (78 pixels per centimeter) atthe object plane with sufficient depth of focus to allow for differencesin x direction position or tilt of the price tags relative to the camerais desirable. For smaller barcodes and 2D barcodes, a higher resolutionmay be appropriate. A digital camera mounted to a support frame 46 sothat it can be relatively stationary while capturing images is thus moresuited to this task than a hand-held smartphone camera or inexpensivewebcams, unless the acquisition is performed close up (e.g., one barcodeat a time with the camera placed very close to the barcode) and thecamera is held sufficiently steady. Furthermore, although the locationsof price tags are somewhat systematic, there are large variations fromshelf to shelf, store to store, and chain to chain, as well asdifferences in lighting conditions, print quality, transparency of theproduct label mounting device 50 (if it overlays the product label 18),and so forth. Thus, it may be appropriate to change the design and/oradjust the configuration of the cameras, depending on the expectedconditions within the store or portion thereof. An exemplary imagecapture assembly 22 is adaptable to accept different numbers of camerasand/or different camera capabilities, as described in further detailbelow.

The exemplary control unit 24 provides both control of the system anddata processing. The control unit 24 includes one or more dedicated orgeneral purpose computing devices configured for performing the methoddescribed in FIG. 7. The computing device may be a PC, such as adesktop, a laptop, palmtop computer, portable digital assistant (PDA),server computer, cellular telephone, tablet computer, pager, combinationthereof, or other computing device capable of executing instructions forperforming the exemplary method. As will be appreciated, although thecontrol unit 24 is illustrated as being physically located on the mobilebase 20 (FIG. 1), it is to be appreciated that parts of the control unitmay be in the image capture assembly 22 or located on a separatecomputer remote from the mobile base and image capture assembly.

The control unit 24 illustrated in FIG. 5 includes a processor 60, whichcontrols the overall operation of the control unit 24 by execution ofprocessing instructions which are stored in memory 62 communicativelyconnected with the processor 60. One or more input/output interfaces 64,66 allow the control unit to communicate (wired or wirelessly) withexternal devices. For example, interface 64 communicates with cameras42, 44, 46 to request image capture, and/or adjustments to the PTZsettings, and to receive captured digital images from the cameras; withtranslation stage 48, where present, to adjust camera position(s); withmobile base 20 for movement of the system as a whole, relative to theshelf unit, and the like. Interface 66 may be used for outputtingacquired or processed images, a store profile 12, and/or informationextracted therefrom, such as to an external computing device and/or aprinter (not shown) for printing and/or packaging sale signage in anappropriate order to match the store profile.

The various hardware components 60, 62, 64, 66 of the control unit 24may be all connected by a bus 68.

The memory 62 may represent any type of non-transitory computer readablemedium such as random access memory (RAM), read only memory (ROM),magnetic disk or tape, optical disk, flash memory, or holographicmemory. In one embodiment, the memory 62 comprises a combination ofrandom access memory and read only memory. In some embodiments, theprocessor 60 and memory 62 may be combined in a single chip. Theinterface 66, 68 allows the computer to communicate with other devicesvia a wired or wireless links or by a computer network, such as a localarea network (LAN) or wide area network (WAN), or the internet, and maycomprise a modulator/demodulator (MODEM), an electrical socket, arouter, a cable, and and/or Ethernet port. Memory 62 stores instructionsfor performing the exemplary method as well as the processed data 12.

The digital processor 60 can be variously embodied, such as by asingle-core processor, a dual-core processor (or more generally by amultiple-core processor), a digital processor and cooperating mathcoprocessor, a digital controller, or the like. The digital processor60, in addition to controlling the operation of the computer 62,executes instructions stored in memory 62 for performing the methodoutlined in FIG. 7 and/or 11.

The term “software,” as used herein, is intended to encompass anycollection or set of instructions executable by a computer or otherdigital system so as to configure the computer or other digital systemto perform the task that is the intent of the software. The term“software” as used herein is intended to encompass such instructionsstored in storage medium such as RAM, a hard disk, optical disk, or soforth, and is also intended to encompass so-called “firmware” that issoftware stored on a ROM or so forth. Such software may be organized invarious ways, and may include software components organized aslibraries, Internet-based programs stored on a remote server or soforth, source code, interpretive code, object code, directly executablecode, and so forth. It is contemplated that the software may invokesystem-level code or calls to other software residing on a server orother location to perform certain functions.

The processor 60 executes instructions 70 stored in memory 62 forperforming the method outlined in FIG. 7 and/or 11. In the illustratedembodiment, the instructions include a configuration component 74, amission planner 76, a translation stage controller 78, a cameracontroller 80, an image data processing component 82, a product datarecognition component 84, a store profile generator 86, and a signagegenerator 88. Fewer than all these components may be included in someembodiments. In other embodiments, some or all of the components may belocated on a separate computing device, i.e., one which is not carriedby the mobile base, as discussed above.

The configuration component 74 is used prior to a mission to configurethe image capture assembly 22 (e.g., determine FOV and position(s) ofthe camera(s) and to provide a spatial characterization of the imagecapture assembly, such as a spatial profile for each camera. Each cameramay have at least one camera spatial profile. A camera may have two ormore spatial profiles if the camera is to be moved, relative to themobile base, and/or its FOV adjusted, for acquiring more than one imageat the same mobile base location. The camera spatial profile may be amapping between pixel location and a location in an x, z plane to enablea mapping between pixels of each image captured at a respective cameraposition and a position in the x, z plane corresponding to a portion ofa shelf face where the images are captured.

The mission planner 76 has access to a store floor plan 90 (layout ofaisle and shelves and its facing) and the purpose of each mission. Amission may be for example, to capture all price tags throughout thestore, or limited to only a part of the store, etc. Using theinformation in the store floor plan 90, the mission planner determinesthe path that the mobile base 20 should follow and communicates with themobile base to provide the path and appropriate stop positions (wherethe images should be acquired by the image capture assembly). Theinstructions may be provided to the mobile base in a step-by-stepfashion or in the form of a full mission.

The translation stage controller 78 determines the translations of thetranslation stage to achieve desired camera positions and communicatesthem to the translation stage 48. The camera controller 80 determinesthe camera parameters (e.g., shutter speed, aperture, ISO number, focallength, . . . ) and optionally position parameters (e.g., pan, tilt,zoom, or vertical translation amount . . . ) of the cameras in the imagecapture assembly for each position that requires image acquisition.These parameters may be fixed throughout the mission and/or adjusteddynamically based on current location information of the mobile base(e.g., distance to the shelf to be imaged, the facing angle, height ofthe shelf . . . ). As will be appreciated, translation stage controller78 and camera controller 80 may form parts of a single component forcontrolling the acquisition of images by the image capture assembly 22.

The image data processing component 82 processes the images acquired byall the cameras and uses the mapping provided by the configurationcomponent and position information provided by the mobile base to mappixels of the captured image to locations in 3D space.

The product data recognition component 84, which may be a part of theimage data processing component 82, analyses the processed images fordetecting price tag locations, extracting product data 26, such as pricetag data, and performs image coordinate conversion (from pixel positionto real-world coordinates).

Outputs of the data processing component 82 and/or product datarecognition component 84 may be used by the store profile generator 88to determine the store profile 12 (e.g., the real-world coordinates ofdetected and recognized UPC codes). In some cases, outputs of the dataprocessing component 82 and/or product data recognition component 84 areused by the translation stage controller 78 and/or camera controller 80to determine what should be the appropriate camera parameters and/orposition parameters for the next image capture. Some outputs of the dataprocessing component 82 and/or product data recognition component 84 maybe used by the mission planner 76 to determine the next positional movefor the mobile base 20.

With reference now to FIG. 7, a method for generating (and using) astore profile 12 is shown, which can be performed with the system ofFIGS. 1-5. As will be appreciated, some or all of the steps of themethod may be performed at least partially manually and need not beperformed in the order described. The method begins at S100.

At S102, the image capture assembly 22 is configured. Briefly, theconfiguration component 74 identifies suitable positions for the cameras42, 44, 46, and optionally a suitable range of camera parameters (e.g.,field of view, exposure time, ISO number, etc.), in order to capture thefull height h of each shelf unit face from a set of overlapping imagesacquired at one single position of the moveable base (i.e., without gapsin the z direction). The configuration component 74 optionally extractsinformation from test images which enables it to associate each (orsome) pixels of a captured image with a point in yz space and/or togenerate a spatial characterization of the image capture assembly whichmay include a spatial profile for each camera.

At S104, a route for scanning the store shelves is computed. Inparticular, the mission planner 76 computes a route for the mobile basearound the facility, based on a store floor plan 90. The floor planidentifies obstructions, particularly locations of shelf units. Thestore plan may have been generated partially automatically, from a priortraversal of the facility by the system 10, for identifying the locationof obstructions. For example, as shown in FIG. 8, the obstructions maybe identified on the floor plan 90 and locations of scannable faces 92on each shelf unit identified (e.g., by a person familiar with thestore). The mission planner 76 computes a route 94, which includes allthe faces 92 and designates parts of the route as a scan path 96 (whereimages of scannable faces 92 are to be acquired) and parts of the routeas a no-scan path 98 (where no images are to be acquired).

At S106, the mission planner 76 communicates the computed route 94 tothe navigation component 30 of the mobile base, and optionallydesignating stop positions, which may be located at approximately equalintervals along the scan path 96. During the mission, the missionplanner 76 receives information from the navigation component 30 fromwhich any deviations to the planned route are computed. The mobile base20 is then responsible for navigating the system 10 to a desiredlocation with desired facing (orientation) requested by the control unit24 and reporting back the actual location and facing if there is anydeviation from the request.

At S108, as the mobile base 20 traverses the route 94, instructions areprovided to the translation stage 48 at each predetermined stop on thescan path 96 for positioning the cameras. The translation stagecontroller 78 communicates instructions to the translation stage 48 whenthe camera position(s) is/are to be adjusted and may provide thetranslation stage 48 with directions for achieving predetermined camerapositions, based on the information generated by the configurationcomponent 74.

At S110, at each predetermined stop on the scan path 96, instructionsare provided to the cameras 40, 42, 44 themselves for positioning andimage acquisition. In particular, the camera controller 80 communicatesinstructions for adjusting position and/or focal plane to the camera'sPTZ components and provides instructions for data acquisition to providethe optimal coverage of the shelf, using the position informationidentified by the configuration component 74. The translation stagecontroller 78 and camera controller 80 may work in cooperation toachieve desired positions of the cameras.

At S112 images 100, 102, are acquired by the cameras at a given positionof the mobile base. The image capture assembly (iteratively) acquiresimages based on the requests by the control unit and the cameraparameters and (optionally) position parameters provided.

At S114, the acquired images 100, 102 are transferred from the cameramemory to the data processing component 82. The data processingcomponent 82 receives the images acquired by the cameras and stores themin memory, such as memory 62, and may perform preliminary processing,such as adjustments for blur, color, brightness, etc. A composite imageor panorama of the shelf face may be computed by performing a union ofmultiple images captured by the image capture assembly. In forming thecomposite image, pixels of one or more of the acquired images may betranslated to account for each camera's spatial profile.

At S116, the product data recognition component 84 processes theacquired images 100, 102 or panorama to identify product data 26 fromthe captured shelf labels 18, where present, in the images. In anexemplary embodiment, the acquired images and a corresponding coarselocation and facing information are analyzed to determine the productlayout information (e.g., via barcode recognition of price tags andknowledge of the camera spatial profile).

The process repeats until the mission is completed (e.g., all aisles ofinterest have been scanned). For a typical mission, the mobile basemoves along each store aisle to enable images of the scannable faces ofeach shelf unit to be captured. From the captured images, each shelfprice tag is detected and its location determined within the image.

By measuring the mobile base's current position in the store floor plan,its position data can then be associated with the images being capturedat that position, based on the time of capture. Candidate regions ofeach image 100, 102 which have at least a threshold probability ofincluding a barcode 54 are identified and processed to extract thebarcode information, which may be output as an SKU code which uniquelyidentifies the product. Associated information, such as price andproduct information 56, 58, particular colors used in the product label18, and the like, may also be used to locate the barcode and/or todecipher it, particularly where the product data recognition componenthas difficulty in doing so based on the barcode alone. The location ofthe barcode in three dimensional space can be determined based on thelocation of the mobile base at the time the image was captured and thespatial characterization of the image capture assembly.

At S118, a store profile 12 is generated based on the identified barcodeinformation 26 and computed barcode locations. In particular, the storeprofile generator 86 generates a store profile 12 which identifieslocations of the price tags 18, based on the extracted barcodeinformation and optionally information provided by one or more of theconfiguration component 74, mission planner 76, and navigation component30, through which pixels of identified barcodes in the captured imagesare associated with a point in real (xyz or xy) space or otherwisegenerally located with respect to the store floor plan 90. An accuratestore profile 12 identifying product locations/locations of price tagsin a store can thus be reconstructed.

At S120, the store profile 12 may be output from the system.

At S122, information on signage to be mounted throughout the store maybe received and a packaging order for the particular store computed,based on the store profile 12. In particular, the signage generator 88receives information on signage to be printed for an upcoming sale inwhich only some but not all of the price tags may need to be replaced.The signage generator uses the store profile 12 to identify thelocations of only the price tags/products to which the sale relates.From this information, a printing and/or packaging order for the signageis generated. When the signage is packaged and provided to an employee,the order in which the signage is packed in accordance with the computedprinting and/or packaging order enables the person to traverse the storein the order in which the signage is packaged to replace/add the newsignage, generally in a single pass through the store. The route definedby the packing order minimizes the amount of backtracking the employeeneeds to do and/or provides for a shorter path (in time or distance) tocomplete the task than would be achievable without the computedstore-specific packaging order, and avoids the need for the store toresort the signage into an appropriate order. In this way, for eachstore in a chain, a store profile can be generated (e.g., periodically),allowing a store-specific packaging order for signage to be computedeach time a set of shelf labels 18 and/or other signage is to be mountedthroughout the store.

The method ends at S124.

Further details of the system and method will now be described.

While in one embodiment, the store profile 12 is used for defining anappropriate sequence for printing/packaging of sale signage, the storeprofile has other applications, including validating that the storeproduct layout complies with a pre-defined planogram. A planogram is apredefined product layout for a slice of about 0.5 meters or more oflength along an aisle. The captured images can also be processed toextract any 1D or 2D barcodes and/or text data from regions that complywith the price tag format. Data such as the product UPC and the pricetag location within the image are extracted.

According to one aspect of this disclosure, provided is a method andsystem that goes beyond typical 1D barcode recognition to improve therecognition rate of identifying product labels, such as a Stock KeepingUnit (SKU) on shelf price-tags. The improvement is gained by utilizingsub-image manipulation and other, i.e., auxiliary, product informationextraction. The method and system has broad usage in retailapplications, including Shelf-Product Identification methods and systemsas previously described with reference to FIGS. 1-7 above. See also U.S.patent application Ser. No. ______, filed ______, by Wu et al., andentitled “Store Shelf Imaging System”. An exemplary method includes thefollowing steps: (1) Acquire image(s) of the shelf in a store; (2)Detect/localize candidate barcode regions on the images; (3) Crop andmanipulate each sub-image/candidate-barcode-region via standard imageprocessing techniques to create modified versions of it; (4) Performbarcode recognition on modified versions of candidate-barcode-regions;(5) For those candidate-barcode-regions, where there is no successfulbarcode recognition yielded on all modified versions of them, determineand detect associated regions (referred as auxiliary product informationregions) corresponding to regions with auxiliary product informationbased on the price-tag layout information; (6) Extract auxiliary productinformation on detected auxiliary product information regions; and (7)Determine possible product information, i.e., SKU, based on acombination of barcode recognition and auxiliary product informationextraction.

Further details of the method and system for recognizing barcodes and/orproduct labels S116 will now be described with reference to FIGS. 8-11.

Acquire image(s) of the shelf in a store. S202

The disclosed method starts with acquired image(s) of the shelf in aretail store. As the images are acquired, they are processed foridentifying the SKU of shelf-products. An example imaging system isshown in FIG. 8 and a high-level use case is further described in U.S.patent application Ser. No. ______, filed ______, by Wu et al., andentitled “Store Shelf Imaging System”, as well as above.

Detect/localize candidate barcode regions on the images. S204

In this step, image analysis is performed on the acquired image todetect and localize candidate barcode regions for further processing.For example, see J. Liyanage, “Efficient Decoding of Blurred, Pitched,and Scratched Barcode Images,” Second International Conference onIndustrial and Information Systems (ICIIS 2007), August, (2007).According to the exemplary embodiment, a combination of average edgestrength, average edge orientation, and morphological filtering are usedfor blob detection. The algorithm is biased to allow morefalse-positives, which will be removed in later processes, whilepenalizing false-negatives more severely. FIG. 9 shows the results ofapplying the algorithm on an example shelf image. As shown in FIG. 9,more false-positives 302, 316, 318, 324, 326, 328 and 330 are detectedto ensure the detection of true positives 310, 312, 314, 320 and 326.These false-positives are easily removed during the barcode recognitionstep.

FIG. 10 shows an example embodiment of candidate 1-D barcode regiondetection S204 via edge properties. As an example, let I(x,y) representthe image intensity, e.g., gray or R or G or B, at position (x,y) S402.First, the horizontal edge map, I_(x)(x,y)=I(x+1,y)−I(x−1,y) is computedS404. This can be achieved efficiently via a convolution operationbetween I(x,y) and a kernel [1−1]. Other gradient type kernel orientedhorizontally can be used as well. Similarly, the vertical edge map iscomputed, I_(y)(x,y)=I(x,y+1)−I(x,y−1) S406. This can be achievedefficiently via a convolution operation between I(x,y) and a kernel[1−1]^(T). Other gradient type kernel oriented vertically can be used aswell. Given the horizontal and vertical edge maps of the images, thecandidate 1-D barcode regions, i.e., a set of regions with sufficientnumber of line strength, can then be detected using the followingexample approach.

First, an initial binary map is generated S408 indicating regions withsufficient edge strength and preferred orientation. As an example, thefollowing rule can be applied for detection of 1-D barcode orientedhorizontally, i.e., regions that may include severalvertical-lines/strong-horizontal-edges:

${B\left( {x,y} \right)} = \left\{ {\begin{matrix}1 & {{{{{{if}\mspace{14mu} {I_{x}\left( {x,y} \right)}} \geq \eta_{1}}\&}{{I_{x}\left( {x,y} \right)}/{I_{y}\left( {x,y} \right)}}} \geq \eta_{2}} \\0 & {otherwise}\end{matrix}.} \right.$

Similarly, the following rule can be applied for detection of 1-Dbarcode oriented vertically, i.e., regions that may include severalhorizontal-lines/strong-vertical-edges:

${B\left( {x,y} \right)} = \left\{ {\begin{matrix}1 & {{{{{{if}\mspace{14mu} {I_{y}\left( {x,y} \right)}} \geq \eta_{1}}\&}{{I_{y}\left( {x,y} \right)}/{I_{x}\left( {x,y} \right)}}} \geq \eta_{2}} \\0 & {otherwise}\end{matrix}.} \right.$

To remove spurious noises in the binarization due to imperfect imaging,the binary map B(x,y) is refined using morphological filtering such asdilation or erosion to yield a better map B′(x,y) S410. The regionswhere B′(x,y)=1 are regions that are likely to have 1-D barcode based onthe strength of edges and their orientation. It is thus possible todetect candidate 1-D barcode regions S412, R_(i) S414, usingconnected-component analyses to determine which set of pixels belong tothe same region, size thresholding to keep only the regions that arewithin certain size ranges based on the expected size of barcode, etc.Note that the resulting number of regions, R_(i), is image dependent,where some may have many while some may have none, and η-dependent.Since it is preferred to have false-positives over havingfalse-negatives (misses) as explained earlier, typically, smaller valuesare chosen for η₁ & η₂.

Note that other edge-based methods or pattern matching methods can beapplied here to detect candidate barcode region as well. Also, theexemplary embodiment described herein uses 1-D barcode detection as anexample, though it can be easily extended for 2-D barcode as well. Forexample, the use of orientation of edges could be removed and replacedwith a compactness measure in two-dimensions if the goal is detecting2-D barcodes rather than 1-D barcodes.

Crop and manipulate each sub-image/candidate-barcode-region via standardimage processing techniques to create modified versions of it. S208

For each detected candidate-barcode-region S204, perform standard imageprocessing techniques to create modified versions of the detectedcandidate-barcode-region. Appropriate image manipulations may includecontrast enhancement, tone-curve reshaping, e.g., gamma-correction,sharpening, de-blurring, morphological filtering especially erosion,etc. See Charles A. Poynton (2003). Digital Video and HDTV: Algorithmsand Interfaces. Morgan Kaufmann. pp. 260, 630. ISBN 1-55860-792-7;Charles Poynton (2010). Frequently Questioned Answers about Gamma; ErikReinhard, Wolfgang Heidrich, Paul Debevec, Sumanta Pattanaik, Greg Ward,and Karol Myszkowski (2010). High Dynamic Range Imaging: Acquisition,Display, and Image-Based Lighting. Morgan Kaufmann. p. 82. ISBN9780080957111; McKesson, Jason L. “Linearity and Gamma—Chapter 12.Dynamic Range”. Learning Modern 3D Graphics Programming. Retrieved 11Jul. 2013; and Gonzalez and Woods, “Digital Image Processing”, 3^(rd)Edition, 954 pages. The objective of this step is to improve “imagequality” so that the barcode recognition rate increases. In prior arts,this step is typically used to manipulate the entire image, and theimage processing is typically selected intelligently based on someadditional analysis or modeling. That approach is ideal if theroot-cause of image degradation is somewhat known or identifiable. Thisis often not feasible in practice. Alternatively, some prior arts useadaptive thresholding in detecting candidate barcode region to alleviatethe impact of poor quality imaging. However, it has been found thatthese are not enough in some applications to solve some practicalissues. Fortunately, for many barcode recognition tasks, the outcome isfairly binary, i.e., recognized or not recognized. When a barcode isrecognized, the probability that the decoded information is incorrect isvery low due to the use of checksum and discrete widths/intensities. Onthe other hand, an algorithm is prone to miss barcodes when the detectedcandidate barcode region does not meet the encoded requirement, i.e.,checksum rules and available discrete widths. Based on the binary natureassociated with barcode recognition, the method disclosed hereinmanipulates the candidate-barcode-regions in an uninformed way, i.e.,not intelligently, while covering a broad range of manipulation space.The barcode recognition step then serves as the selector knowing that ifa barcode is recognized, it is most likely that the decoded informationis correct and indeed there is a barcode at thatcandidate-barcode-region. Importantly, this method only works for abinary task that is biased to high accuracy once recognized. Forexample, an optical character recognition (OCR) process will achieveless accurate results, since an “O” can be misread as “0”, “8”, etc., ifinappropriate image manipulation is applied. Clearly, this step can beapplied on the entire image rather than only candidate-barcode-regions.However, given a limited amount of computation resources, the methodapplies an order or more (10×) manipulations on allcandidate-barcode-regions rather than those on an entire image. Thisassumes that the barcode detection step can perform well without theseadditional image manipulations and the entire image is not filled withbarcodes. Both assumptions are true for the application of the disclosedbarcode recognition method to product labels as described herein. Notethat the original sub-image is retained as one of the “modified”versions for later processing, and the image manipulation and barcoderecognition of each candidate-barcode-region can be performed in asequential or a parallel fashion. For sequential processing, one imagemanipulation is performed on a candidate barcode-region which is thenpassed to the barcode recognition step S210. If the modified version isrecognized as including a barcode, the sequential process stops. If not,the sequential process continues until a barcode is recognized or untilall image manipulations processes have been tested without success. Fora parallel process, all image manipulations are performed on acandidate-barcode-region first, and all of the modified versions arepassed to the barcode recognition process 210. Finally, the recognitionresults are the union of the individual results. The latter approach hasthe disadvantage of some waste in computation, but has the advantage ofa simpler system architecture. Since all processes, i.e., imagemanipulation and barcode recognition, are done at sub-image levels thewaste is negligible in practice.

Perform barcode recognition on modified versions ofcandidate-barcode-regions. S210

In this step, barcode recognition is performed on the modified versionsof sub-images for each candidate-barcode-region using conventionalbarcode recognition algorithm(s). The final barcode recognition resultfor each candidate-barcode-region is the fusion of all results on itscorresponding modified versions. One fusion method is to use the unionof all results. For example, assume three modified versions wererecognized as including the barcodes 0123456 and 0123458, and the thirdbarcode was not recognized. Then, the final result is: recognized with0123456 and recognized with 0123458. Based on an implementation of anexemplary barcode recognition method, most of the time, at most onebarcode is recognized for each modified version; and if recognized theyall have the same decoded information. In rare cases, where more thanone barcode is recognized for one candidate-barcode-region, the unionoperation keeps all of them. Another fusion method is to simply keep thebarcode with the highest confidence score. According to the exemplaryembodiment, the fusion method is utilized since tests indicated that thebarcode detection disclosed herein rarely detects more than one barcodein one candidate-barcode-region. Also, some candidate-barcode-regionshave no barcode detected due to the false-positive bias imposed on thebarcode detection algorithm.

For those candidate-barcode-regions where there is no successful barcoderecognition yielded on all modified versions of them, determine anddetect associated regions, referred as auxiliary product informationregions, corresponding to regions with auxiliary product informationbased on the price-tag layout information S214.

In this step, determine and detect associated regions, referred to asother/auxiliary product information regions, corresponding to regionswith auxiliary product information based on the price-tag layoutinformation. This step is performed only for thosecandidate-barcode-regions where there is no successful barcoderecognition yielded on all modified versions of thecandidate-barcode-regions. The basic idea is as follows. For mostprice-tags in a retail environment, there is additional productinformation beyond the machine readable barcode. The additional productinformation can be helpful to narrow down possible SKU's even if thebarcode recognition fails for the particular price-tag. Notably, in atypical retail setting it is much easier to detect and recognize barcodethan to detect auxiliary product information regions and decipher thecorresponding product information. Furthermore, barcode information ismostly unique while the additional product information may not be.Nonetheless, the detection of additional product information can behelpful if feasible. Fortunately, for a typical retail store there areonly a minimum number of price-tag templates. Therefore, it is easier todetect auxiliary product information regions on the price-tag once itsbarcode-region is detected. Additionally, a skew/rotation correction canbe applied to these detected auxiliary product information regions basedon an estimated orientation of the corresponding detectedbarcode-region. FIG. 11 shows example price-tag layouts 420, 422, 424and 426 commonly seen in retail store. Common elements are: product SKUin the form of 1D barcode (EAN-13, UPC) and in the form of text, price(text), product information (text). For on-sales tag, additionalinformation in the form of text such as “Save XX¢”, “NEW PRICE”, “ENDSAPRIL 22”, “SALE”, . . . is also quite common; and there is oftensaturated color patch (red, yellow, etc.) on the on-sales price-tags.These are all good candidate auxiliary product information regions forthis detection. A limited number of price-tag templates available makesit feasible to determine and detect these regions robustly.

Extract auxiliary product information on detected auxiliary productinformation regions. S216

In this step, auxiliary product information is extracted from thosedetected auxiliary product information regions. The extractionalgorithms are price-tag dependent while each individual extractionalgorithm is mostly known and available. Hence, one of the tasks of thisstep is to select appropriate extraction algorithms for a given set ofpossible price-tag layout for analyzing these auxiliary productinformation regions. Below are a few example/typical extractionalgorithms for this step:

Optical Character Recognition (OCR) for extracting text information

Color patch detection for sales-tag detection (red-tag, yellow-tag,etc.)

Logo detection for extracting informative image element such as logo,thumbnail, marker etc.

Template matching for extracting keywords in image form such as “SAVE”,“SALE”, “NEW PRICE”, etc.

Clearly, a very capable OCR engine is available, the entire price-tagregion can be inputted to the OCR engine to extract alltext-information. However, this has been found to be not effective,i.e., either not accurate enough or too computational expensive. Theexemplary embodiment provided herein identifies text-form sub-elements,e.g., text form of SKU information, price, “SAVE XX¢”, in isolation andruns OCR engines on each text-form sub-element instead. This helps intwo ways: (1) it simplifies the segmentation process during OCR and (2)it allows the OCR to run in an informed manner. It has been found thatrunning an OCR engine in a single text line mode performs better than indocument mode, which can have multiple lines of text. It has also beenfound that when performing OCR on the text form of SKU information, theperformance is better if the OCR engine knows that only “digits” areallowed in the given line of text.

Determine possible SKU information based on a combination of barcoderecognition and auxiliary product information extraction. S218

In this step, the final SKU information is determined based on acombination of the barcode recognition and auxiliary product informationextraction for each candidate-barcode-region. For eachcandidate-barcode-region, if at least one barcode is recognized, thefinal result is from the barcode recognition step, i.e., detectedlocation and decoded information, and the entire auxiliary productinformation extraction process is skipped. Note that the SKU informationunder this situation is very accurate as discussed before.Alternatively, if there is no barcode recognized for thecandidate-barcode-region, this step uses all extracted auxiliary productinformation to narrow down possible SKU information and use that as thefinal estimated SKU information associated with thecandidate-barcode-region. The accuracy under this situation is highlyvariable. Hence, also provided is a warning label and confidence scoreso that the users of this information are aware of the potential risk.For example, if in one of the missions of detecting all SKU informationof a store, there is only one on-sales price-tag whose SKU is notdetected by barcode recognition and our auxiliary product informationextraction process concludes that a given candidate-barcode-region is anon-sale tag, then it is certain that the SKU must be the one that ismissing from barcode recognition. However, if there are more than oneunclaimed SKUs, then the use of an OCR result on the text-form of SKUinformation regions may be needed to further narrow the possibilities.The process can go even deeper depending on the practical situations ofthe application.

Notable differences of the disclosed method and system compared to theprior art are twofold: (1) use of broad-range image manipulations atsub-image levels and (2) use of auxiliary product informationextraction. By performing image manipulations at sub-image levels, alarge number of image manipulations (over-sampled trial-and-errorstrategy) are able to be applied and improvements are gained in thebarcode recognition rate while keeping the additional computationsrelatively low. By using the layout knowledge of price-tags in a store,the method and system disclosed is able to better localize auxiliaryproduct information and extract the product information that helpsnarrow down the possible SKU information where barcode recognition failscompletely. Note that strategy (1) or strategy (2) can also be usedindependent of each other.

Some portions of the detailed description herein are presented in termsof algorithms and symbolic representations of operations on data bitsperformed by conventional computer components, including a centralprocessing unit (CPU), memory storage devices for the CPU, and connecteddisplay devices. These algorithmic descriptions and representations arethe means used by those skilled in the data processing arts to mosteffectively convey the substance of their work to others skilled in theart. An algorithm is generally perceived as a self-consistent sequenceof steps leading to a desired result. The steps are those requiringphysical manipulations of physical quantities. Usually, though notnecessarily, these quantities take the form of electrical or magneticsignals capable of being stored, transferred, combined, compared, andotherwise manipulated. It has proven convenient at times, principallyfor reasons of common usage, to refer to these signals as bits, values,elements, symbols, characters, terms, numbers, or the like.

It should be understood, however, that all of these and similar termsare to be associated with the appropriate physical quantities and aremerely convenient labels applied to these quantities. Unlessspecifically stated otherwise, as apparent from the discussion herein,it is appreciated that throughout the description, discussions utilizingterms such as “processing” or “computing” or “calculating” or“determining” or “displaying” or the like, refer to the action andprocesses of a computer system, or similar electronic computing device,that manipulates and transforms data represented as physical(electronic) quantities within the computer system's registers andmemories into other data similarly represented as physical quantitieswithin the computer system memories or registers or other suchinformation storage, transmission or display devices.

The exemplary embodiment also relates to an apparatus for performing theoperations discussed herein. This apparatus may be specially constructedfor the required purposes, or it may comprise a general-purpose computerselectively activated or reconfigured by a computer program stored inthe computer. Such a computer program may be stored in a computerreadable storage medium, such as, but is not limited to, any type ofdisk including floppy disks, optical disks, CD-ROMs, andmagnetic-optical disks, read-only memories (ROMs), random accessmemories (RAMs), EPROMs, EEPROMs, magnetic or optical cards, or any typeof media suitable for storing electronic instructions, and each coupledto a computer system bus.

The algorithms and displays presented herein are not inherently relatedto any particular computer or other apparatus. Various general-purposesystems may be used with programs in accordance with the teachingsherein, or it may prove convenient to construct more specializedapparatus to perform the methods described herein. The structure for avariety of these systems is apparent from the description above. Inaddition, the exemplary embodiment is not described with reference toany particular programming language. It will be appreciated that avariety of programming languages may be used to implement the teachingsof the exemplary embodiment as described herein.

A machine-readable medium includes any mechanism for storing ortransmitting information in a form readable by a machine, e.g., acomputer. For instance, a machine-readable medium includes read onlymemory (“ROM”); random access memory (“RAM”); magnetic disk storagemedia; optical storage media; flash memory devices; and electrical,optical, acoustical or other form of propagated signals, e.g., carrierwaves, infrared signals, digital signals, etc., just to mention a fewexamples.

The methods illustrated throughout the specification, may be implementedin a computer program product that may be executed on a computer. Thecomputer program product may comprise a non-transitory computer-readablerecording medium on which a control program is recorded, such as a disk,hard drive, or the like. Common forms of non-transitorycomputer-readable media include, for example, floppy disks, flexibledisks, hard disks, magnetic tape, or any other magnetic storage medium,CD-ROM, DVD, or any other optical medium, a RAM, a PROM, an EPROM, aFLASH-EPROM, or other memory chip or cartridge, or any other tangiblemedium from which a computer can read and use.

Alternatively, the method may be implemented in transitory media, suchas a transmittable carrier wave in which the control program is embodiedas a data signal using transmission media, such as acoustic or lightwaves, such as those generated during radio wave and infrared datacommunications, and the like.

It will be appreciated that variants of the above-disclosed and otherfeatures and functions, or alternatives thereof, may be combined intomany other different systems or applications. Various presentlyunforeseen or unanticipated alternatives, modifications, variations orimprovements therein may be subsequently made by those skilled in theart which are also intended to be encompassed by the following claims.

What is claimed is:
 1. A method of performing decoding of a barcodeassociated with a product label, the product label including one or morebarcodes and other product label information, the method comprising: animage capturing device capturing an image of the product label andstoring the captured image in a memory operatively associated with theimage capturing device; a processor operatively associated with thememory detecting and localizing one or more barcode candidate regionswithin the captured image of the product label, the barcode candidateregions including a substantially fewer number of pixels relative to atotal number of pixels included in the captured pixel image of theproduct label; cropping the detected and localized one or more barcodecandidate regions to generate one or more sub-images including images ofthe one or more barcode candidate regions; processing each sub-imageusing two or more independent image quality improvement processes togenerate modified versions of the barcode candidate regions; andprocessing the modified versions of the barcode candidate regions todecode readable barcodes included in the modified versions of thecandidate barcode regions.
 2. The method of performing decoding of abarcode according to claim 1, wherein the two or more independent imagequality improvement processes include one or more of contrastenhancement, tone-curve reshaping, gamma-correction, sharpening,de-blurring and morphological filtering.
 3. The method of performingdecoding of a barcode according to claim 1, wherein the two or moreindependent image quality improvement processes are performed in serialor parallel.
 4. The method of performing decoding of a barcodeassociated with a product label according to claim 1, wherein thebarcode is one of a 1D (Dimensional) and 2D barcode.
 5. The method ofperforming decoding of a barcode associated with a product labelaccording to claim 1, further comprising: associating the product labelincluding the decoded barcode with a location in a retail environment.6. The method of performing decoding of a barcode associated with aproduct label according to claim 5, wherein the step of detecting andlocalizing one or more barcode-candidate-regions is biased to generaterelatively more false-positives than false-negatives.
 7. The method ofperforming decoding of a barcode associated with a product labelaccording to claim 1, further comprising: fusing the decoded readablebarcodes and identifying a product label based on the fusion.
 8. Themethod of performing decoding of a barcode associated with a productlabel according to claim 1, further comprising: further processing thecaptured image of the product label to extract other non-barcodeinformation to identify the product label if an associatedcandidate-barcode-region includes an unreadable barcode.
 9. An imageprocessing system for performing decoding of a barcode associated with aproduct label, the product label including one or more barcodes andother product label information, the image processing system comprising:an image capturing device; and a processor operatively connected to theimage capturing device, the processor configured to: the image capturingdevice capturing an image of the product label and storing the capturedimage in a memory operatively associated with the image capturingdevice; the processor operatively associated with the memory detectingand localizing one or more barcode candidate regions within the capturedimage of the product label, the barcode candidate regions including asubstantially fewer number of pixels relative to a total number ofpixels included in the captured pixel image of the product label;cropping the detected and localized one or more barcode candidateregions to generate one or more sub-images including images of the oneor more barcode candidate regions; processing each sub-image using twoor more independent image quality improvement processes to generatemodified versions of the barcode candidate regions; and processing themodified versions of the barcode candidate regions to decode readablebarcodes included in the modified versions of the candidate barcoderegions.
 10. The image processing system according to claim 9, whereinthe two or more independent image quality improvement processes includeone or more of contrast enhancement, tone-curve reshaping,gamma-correction, sharpening, de-blurring and morphological filtering.11. The image processing system according to claim 9, wherein the two ormore independent image quality improvement processes are performed inserial or parallel.
 12. The image processing system according to claim9, wherein the barcode is one of a 1D (Dimensional) and 2D barcode. 13.The image processing system according to claim 9, further comprising:associating the product label including the decoded barcode with alocation in a retail environment.
 14. The image processing systemaccording to claim 13, wherein the step of detecting and localizing oneor more barcode-candidate-regions is biased to generate relatively morefalse-positives than false-negatives.
 15. The image processing systemaccording to claim 9, further comprising: fusing the decoded readablebarcodes and identifying a product label based on the fusion.
 16. Theimage processing system according to claim 9, further comprising:further processing the captured image of the product label to extractother non-barcode information to identify the product label if anassociated candidate-barcode-region includes an unreadable barcode. 17.A method of performing product label identification, the product labelincluding one or more barcodes and other product label information, themethod comprising: an image capturing device capturing an image of theproduct label and storing the captured image in a memory operativelyassociated with the image capturing device; a processor operativelyassociated with the memory detecting and localizing one or more barcodecandidate regions within the captured image of the product label;cropping the detected and localized one or more barcode candidateregions to generate one or more sub-images including the one or morebarcode candidate regions; processing each sub-image to decode readablebarcodes included in the barcode candidate regions and identify barcodecandidate regions including an unreadable barcode; processing all or aportion of the captured image of the product label associated with theunreadable barcode to determine all or part of the other product labelinformation association with the unreadable barcode; comparing thedetermined other product label information to a plurality of productlabel templates to determine a layout associated with the product labelincluding an unreadable barcode candidate region; processing thecaptured image of the product label to extract all or part of the otherproduct label information based on the determined layout associated withthe product label; and identifying the captured image of the productlabel as including one of a plurality of unique predefined productlabels.
 18. The method of performing product label identificationaccording to claim 17, wherein an estimated orientation of a barcodecandidate region associated with a nonreadable barcode is used to applya correction to all or part of the captured image of the product label.19. The method of performing product label identification according toclaim 18, wherein the correction is associated with one or more of skewand rotation.
 20. The method of performing product label identificationaccording to claim 17, wherein the step of processing the captured imageof the product label selects and uses one of a plurality of extractionalgorithms based on the determined layout associated with the productlabel.
 21. The method of performing product label identificationaccording to claim 17, wherein the step of identifying the capturedimage of the product label includes associating a relative confidencevalue with the identification of the product label.
 22. The method ofperforming product label identification according to claim 1, whereinthe product label identification includes a SKU (stock keeping unit).23. The method of performing product label identification according toclaim 1, wherein the other product label information includes one ormore SKU text, price text, product information text, logo, and colorpatch.